<!doctype html>
<html lang="en">
<head>
    <meta charset="utf-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <meta name="description" content="coinfaces">

        <title>rsync备份</title>

    <link rel="stylesheet" href="../../theme/css/style.css">
    <link rel="stylesheet" href="../../theme/css/pygments.css">



</head>

<body>
    <div class="content">
<h1>rsync备份</h1>    <p>
        under
            <a href="../../tags/rsync/">rsync</a>
            <a href="../../tags/backup/">backup</a>
    </p>
    <p>
        in <a href="../../categories/tech/">tech</a>
    </p>
    <p>Published: 2016-06-10</p>


    <div class="section" id="id1">
<h2>rsync用法简介</h2>
<p><a class="reference external" href="http://rsync.samba.org/">rsync</a>是一款功能强大的文件同步和备份工具，其突出的一点是很省，通过差异备份尽可能避免不必要的文件传输，配合Linux文件系统的硬链接功能，还能在保持备份高可用的情况下，利用同样大小的备份空间创建尽更多的备份。</p>
<p>使用rsync时要注意源目录末尾是否有“/”会影响rsync的行为，如果没有“/”，rsync会在目标目录下创建一个同名源目录，这点cp命令很不一样。</p>
<p>下面列出了rsync的常用选项，对应的长、短选项请参考rsync文档。</p>
<table class="docutils option-list" frame="void" rules="none">
<col class="option" />
<col class="description" />
<tbody valign="top">
<tr><td class="option-group">
<kbd><span class="option">-a</span></kbd></td>
<td>很常见，基本保证了目标文件的内容和各种属性（包括时间、权限、用户、组等）都和源文件一模一样，这些属性里不包括ACL（-A）和扩展属性（-X）。</td></tr>
<tr><td class="option-group">
<kbd><span class="option">--delete</span></kbd></td>
<td>删除源里没有对应的目标路径</td></tr>
<tr><td class="option-group">
<kbd><span class="option">-H</span></kbd></td>
<td>保持硬链接，如果源文件中两个文件是硬链接，那么备份后创建的相应的备份文件也指向同一个inode。</td></tr>
<tr><td class="option-group">
<kbd><span class="option">--numeric-ids</span></kbd></td>
<td>确保uid和gid在不同用户和组的系统之间保持不变</td></tr>
<tr><td class="option-group" colspan="2">
<kbd><span class="option">--exclude=<var>&lt;pattern&gt;</var></span></kbd></td>
</tr>
<tr><td>&nbsp;</td><td>不备份匹配的源路径，在--delete的情况下，不删除匹配的目标路径</td></tr>
<tr><td class="option-group" colspan="2">
<kbd><span class="option">--exclude-from=<var>&lt;file&gt;</var></span></kbd></td>
</tr>
<tr><td>&nbsp;</td><td>同--exclude，但所有的pattern都写在该文本文件里</td></tr>
</tbody>
</table>
</div>
<div class="section" id="id3">
<h2>排除路径</h2>
<p>下面是常见的需被排除的路径</p>
<ul class="simple">
<li>如果目标目录是源目录的子目录，则应被列在排除路径里，否则以前的备份会被再备份一遍</li>
<li>/dev、/proc、/sys、/run、/var/run、/tmp等只包括运行时文件的目录</li>
<li>lost+found</li>
<li>/mnt、/media等用来挂载外部数据的目录</li>
<li>用bind方式挂载的目录和源目录中确保只有一个会被备份</li>
<li>用户主目录下的.cache、.local/share/Trash、.thumbnails等路径</li>
<li>所有的.gvfs目录</li>
</ul>
<p>如果不想备份某个目录下的文件，但又需要创建该目标目录时，应在目录的路径后加“/<em>”（如--exclude=&quot;/dev/</em>&quot;）。</p>
</div>
<div class="section" id="id4">
<h2>使用硬链接</h2>
<p>假设按备份时间从新到旧有backup.0、backup.1和backup.2三个备份，则一次备份执行的步骤大致如下：</p>
<pre class="literal-block">
rm -rf backup.2
mv backup.1 backup.2
mv backup.0 backup.1
rsync -a --delete --link-dest=backup.1 source/ backup.0/
</pre>
<p>在--link-dest选项可用之前，可以用下面的命令代替最后两步：</p>
<pre class="literal-block">
cp -al backup.0 backup.1
rsync -a --delete source/ backup.0/
</pre>
<p>但这种方法有一个问题，如果在上一次备份之后，有一个文件的属性发生了变化，但内容没变，这种情况下rsync会直接修改backup.0目录里相应文件的属性，从而导致backup.1里文件的属性也被改变，而如果使用--link-dest，由于backup.0是空的，该文件会被再次复制到backup.0里，虽然多占用了备份空间，但确保了旧的备份不发生变化。</p>
<p>注意使用--link-dest最好保持backup.0为空，如果backup.0里有一文件和backup.1里对应的文件不一致，即使backup.1里的文件和source里的文件完全一致，rsync也会用source里的文件覆盖backup.0里的文件，而非将backup.1里的文件链接到backup.0里，因为只有在backup.0里没有对应文件时，rsync才会查看backup.1目录。</p>
</div>
<div class="section" id="id5">
<h2>备份的循环</h2>
<p>最简单情况如上所述，每次新增的备份都占用最小的序号，在此之前递增旧备份的序号将最小的序号空出来，当备份的序号超过预设值时就将其删除。</p>
<p>大多数情况下没必要按新备份的频率来保存旧备份，因此可以让备份按间隔递增的数列分布，如2的指数分布（1、2、4、8、16）或斐波那契分布（1、2、3、5、8）。</p>
<p>按2的指数分布保存备份的算法很简单，假设备份的最大序号是<tt class="docutils literal">2^n</tt>，每次新建备份后，仅保留<tt class="docutils literal"><span class="pre">2^(k-1)+1</span></tt>到<tt class="docutils literal">2^k</tt>之间序号最大的备份（k从2到n）。</p>
<p>用一个比较复杂的算法可以按斐波那契数列分布保存备份。假设理想的备份序列是s，s共n项，在某个时间点的备份序列是a，a有m项（<tt class="docutils literal">m&lt;n</tt>）。令i从m-1到2，如果<tt class="docutils literal"><span class="pre">a[i]-a[i-2]</span> &lt;= <span class="pre">s[i-1]-s[i-2]</span></tt>并且<tt class="docutils literal"><span class="pre">a[j+1]-a[j]</span> &lt;= <span class="pre">s[j]-s[j-1]</span></tt>（j从i到m-2），则删除<tt class="docutils literal"><span class="pre">a[i-1]</span></tt>。这个算法也适用于2的指数分布，目前还没有看到用斐波那契分布保存备份，猜测是没有特别简单的实现方法。</p>
<p>上述两种备份保存示意算法如下：</p>
<div class="highlight"><pre><span></span><span class="kn">import</span> <span class="nn">math</span>


<span class="k">class</span> <span class="nc">BkSeq</span><span class="p">(</span><span class="nb">object</span><span class="p">):</span>
    <span class="sd">&#39;&#39;&#39;Backup Sequence&#39;&#39;&#39;</span>

    <span class="k">def</span> <span class="fm">__init__</span><span class="p">(</span><span class="bp">self</span><span class="p">,</span> <span class="n">init_list</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">standard_list</span><span class="o">=</span><span class="kc">None</span><span class="p">):</span>
        <span class="k">try</span><span class="p">:</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">data</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">init_list</span><span class="p">)</span>
        <span class="k">except</span> <span class="ne">TypeError</span><span class="p">:</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">data</span> <span class="o">=</span> <span class="p">[</span><span class="mi">1</span><span class="p">]</span>

        <span class="k">try</span><span class="p">:</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">standard_list</span> <span class="o">=</span> <span class="nb">list</span><span class="p">(</span><span class="n">standard_list</span><span class="p">)</span>
        <span class="k">except</span> <span class="ne">TypeError</span><span class="p">:</span>
             <span class="bp">self</span><span class="o">.</span><span class="n">standard_list</span> <span class="o">=</span> <span class="kc">None</span>

    <span class="k">def</span> <span class="fm">__repr__</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="k">return</span> <span class="nb">repr</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">)</span>

    <span class="k">def</span> <span class="nf">rotate</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="k">for</span> <span class="n">i</span><span class="p">,</span> <span class="n">x</span> <span class="ow">in</span> <span class="nb">enumerate</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">):</span>
            <span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">+=</span> <span class="mi">1</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="o">.</span><span class="n">insert</span><span class="p">(</span><span class="mi">0</span><span class="p">,</span> <span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">-</span> <span class="mi">1</span><span class="p">)</span>
        <span class="bp">self</span><span class="o">.</span><span class="n">remove</span><span class="p">()</span>


<span class="k">class</span> <span class="nc">ExponentialBkSeq</span><span class="p">(</span><span class="n">BkSeq</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">remove</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">found</span> <span class="o">=</span> <span class="p">{}</span>
        <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">reversed</span><span class="p">(</span><span class="nb">range</span><span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">))):</span>
            <span class="n">m</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">math</span><span class="o">.</span><span class="n">log</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="n">i</span><span class="p">],</span> <span class="mi">2</span><span class="p">))</span>
            <span class="k">if</span> <span class="n">found</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="n">m</span><span class="p">,</span> <span class="kc">False</span><span class="p">):</span>
                <span class="k">del</span> <span class="bp">self</span><span class="o">.</span><span class="n">data</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
            <span class="k">else</span><span class="p">:</span>
                <span class="n">found</span><span class="p">[</span><span class="n">m</span><span class="p">]</span> <span class="o">=</span> <span class="kc">True</span>


<span class="k">class</span> <span class="nc">GeneralBkSeq</span><span class="p">(</span><span class="n">BkSeq</span><span class="p">):</span>
    <span class="k">def</span> <span class="nf">remove</span><span class="p">(</span><span class="bp">self</span><span class="p">):</span>
        <span class="n">L</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">data</span>
        <span class="n">S</span> <span class="o">=</span> <span class="bp">self</span><span class="o">.</span><span class="n">standard_list</span>

        <span class="n">m</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">L</span><span class="p">)</span>
        <span class="n">i</span> <span class="o">=</span> <span class="n">m</span> <span class="o">-</span> <span class="mi">1</span>
        <span class="k">while</span> <span class="n">i</span> <span class="o">&gt;=</span> <span class="mi">2</span><span class="p">:</span>
            <span class="c1"># Remove elements in L that exceed allowed maximum value</span>
            <span class="k">if</span> <span class="n">L</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">&gt;</span> <span class="n">S</span><span class="p">[</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
                <span class="k">del</span> <span class="n">L</span><span class="p">[</span><span class="n">i</span><span class="p">]</span>
                <span class="n">m</span> <span class="o">-=</span> <span class="mi">1</span>
                <span class="n">i</span> <span class="o">=</span> <span class="n">m</span> <span class="o">-</span> <span class="mi">1</span>
                <span class="k">continue</span>

            <span class="c1"># Assume before removal, the differences between elements in L</span>
            <span class="c1"># do not exceed that in S</span>
            <span class="c1"># These checks ensure the same truth holds after the removal of</span>
            <span class="c1"># L[i-1]</span>
            <span class="k">if</span> <span class="n">L</span><span class="p">[</span><span class="n">i</span><span class="p">]</span> <span class="o">-</span> <span class="n">L</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">2</span><span class="p">]</span> <span class="o">&lt;=</span> <span class="n">S</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span> <span class="o">-</span> <span class="n">S</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">2</span><span class="p">]:</span>
                <span class="k">for</span> <span class="n">j</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">i</span><span class="p">,</span> <span class="n">m</span><span class="o">-</span><span class="mi">1</span><span class="p">):</span>
                    <span class="k">if</span> <span class="n">L</span><span class="p">[</span><span class="n">j</span><span class="o">+</span><span class="mi">1</span><span class="p">]</span> <span class="o">-</span> <span class="n">L</span><span class="p">[</span><span class="n">j</span><span class="p">]</span> <span class="o">&gt;</span> <span class="n">S</span><span class="p">[</span><span class="n">j</span><span class="p">]</span> <span class="o">-</span> <span class="n">S</span><span class="p">[</span><span class="n">j</span><span class="o">-</span><span class="mi">1</span><span class="p">]:</span>
                        <span class="k">break</span>
                <span class="k">else</span><span class="p">:</span>
                    <span class="k">del</span> <span class="n">L</span><span class="p">[</span><span class="n">i</span><span class="o">-</span><span class="mi">1</span><span class="p">]</span>
                    <span class="n">m</span> <span class="o">-=</span> <span class="mi">1</span>
                    <span class="n">i</span> <span class="o">=</span> <span class="n">m</span> <span class="o">-</span> <span class="mi">1</span>
                    <span class="k">continue</span> <span class="c1"># while loop</span>

            <span class="n">i</span> <span class="o">-=</span> <span class="mi">1</span>

<span class="k">def</span> <span class="nf">test</span><span class="p">():</span>
    <span class="n">s</span> <span class="o">=</span> <span class="n">ExponentialBkSeq</span><span class="p">()</span>
    <span class="c1">#s = GeneralBkSeq([1], [1, 2, 4, 8, 16, 32, 64, 128])</span>
    <span class="c1">#s = GeneralBkSeq([1], [1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144])</span>
    <span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="mi">150</span><span class="p">):</span>
        <span class="n">s</span><span class="o">.</span><span class="n">rotate</span><span class="p">()</span>
        <span class="nb">print</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>


<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s1">&#39;__main__&#39;</span><span class="p">:</span>
    <span class="n">test</span><span class="p">()</span>
</pre></div>
</div>
<div class="section" id="id6">
<h2>保护备份不被意外覆写</h2>
<p>保护备份的要点是将备份和正在运行的系统尽量分开，最好是放在不同地理位置的一台电脑里，如果条件有限或确实需要让用户能访问备份，也不能让用户拥有写入的权限。</p>
<p>撇除软件的bug，如果要让用户访问备份数据，有这么几种方法：</p>
<ul class="simple">
<li>用<tt class="docutils literal"><span class="pre">--bind</span></tt>参数以只读的方式挂载备份目录（在一些2.4老内核系统里无效）</li>
<li>将备份导出为NFS只读目录，详见<a class="reference external" href="http://www.mikerubel.org/computers/rsync_snapshots/">Mike Rubel的文章</a></li>
<li>用<tt class="docutils literal">chattr +i</tt>命令使备份不可写，需文件系统支持</li>
</ul>
</div>
<div class="section" id="id8">
<h2>参考</h2>
<p>以上内容总结于这三篇文章，<a class="reference external" href="http://www.mikerubel.org/computers/rsync_snapshots/">Mike Rubel的“Easy Automated Snapshot-Style Backups with Linux and Rsync”</a>是较早的一篇总结用rsync和硬链接实现循环快照备份的文章，涉及如硬链接的使用、循环方法、备份的保护等各种细节的取舍，是被引用得较多的一篇文章，<a class="reference external" href="http://www.pointsoftware.ch/howto-local-and-remote-snapshot-backup-using-rsync-with-hard-links/">Francois Scheurer的“Howto – local and remote snapshot backup using rsync with hard links”</a>发展了前一篇文章的方法，加入调整新旧备份保存频率，以及用chattr保护备份数据，<a class="reference external" href="https://wiki.archlinux.org/index.php/full_system_backup_with_rsync">Arch Linux wiki - “Full system backup with rsync”</a>则介绍了rsync备份时常需被排除的备份路径。</p>
<p>（完）</p>
</div>

    </div>
<nav class="nav sidebar">
    <div class="siteinfo">
        <div class="sitename"><a href="../..">coinfaces</a></div>
        <div class="tagline">Another Side of Life</div>
    </div>

        <a href="../../categories/">categories</a>
        <a href="../..">&larr; Home</a>
</nav><footer class="footer">
    <span>
        &copy; jpli &middot;
        under <a rel="license" href="http://creativecommons.org/licenses/by-sa/4.0/">CC-BY-SA 4.0</a>
    </span>
    |
    <span>
        Powered by:
        <a href="http://blog.getpelican.com/">Pelican</a>
    </span>
</footer></body>
</html>